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ABSTRACT 

The paper presents the conceptual framework and 
research strategy of the psychiatric utilization review and 
evaluation (PURE) project at the Connecticut Mental Health Center. 
Review by structure, by outcome and by process are considered 
briefly. The Basic Utilization Review Program was developed to 
provide a more sophisticated and economical analysis of process. The 
author's conception of utilization review encompasses a review of: 

(1) which patients come to the center; (2) what oriorities are 
assigned to patients; (3) what services are provided; and (4) 
outcome. ?our stages in the research proqram are delineated: (1) the 
development of criteria for evaluating patient care; (2) testing the 
criteria for feasibility, reliability and validity; (3) implementing 
the above utilization review mechanism; and (4) exporting the system 
of evaluation and review to other mental health centers. The general 
model is examined. (TL) 
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INTRODUCTION 



The psychiatric utilization review and evaluation (PURE) 
project at the Connecticut Mental Health Center at Yale University 
was undertaken to develop techniques for utilization review and 
patient care evaluation in community mental health centers, and 
to apply these techniques as a major component of program 
evaluation in community mental health. The purpose of this paper 
is to present the conceptual framework employed and the research 
strategy of the project* Discussed will be several of the major 
techniques of medical care evaluation, the special problems of 
applying them to psychiatry, the manner in which the project 
strategy addresses these problems, and finally the integration 
of the project into a general model of program evaluation. 



TYPES OF REVIEW 



Within ^he field of medical care one can, for heuristic 
purpose, separate three types of review, although in fact, they 
are interdependent. They are review by structure, by outcome 
and by process. 

Review of Structure 

In review of structure, attention is directed to adequacy of 
facilities, services and manpower. In review of manpower, it is 
assumed that qualified experts in the delivery of care are necessary, 
or perhaps, necessary and sufficient to insure that quality care is 
given. One investigates the education and prior experience of the 
staff and uses this as an index of their professional competence. 

One could also describe and evaluate the sets of relationships within 
a program, professional time commitments, and organizational hier- 
archies including areas of conflict of interest, in order to determine 
whether there are structural blocks preventing accomplishments of 
program goals. Accreditation and licensure are among the methods 
for carrying out this assessment, 

Review of Outcome 

A fundamentally different kind of approach to review is the 
evaluation of outcome. Some direct measure of end results is 
related to treatment given. Theoretically within this conceptual 
framework the qualifications of those rendering the care, the 
structure of their organization, and the extent to which they use 
acceptable methods can be disregarded in favor of appraising 
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whether or not that which is desired is achieved. For instance, 
have the drug addicts stopped using drugs after treatment? Are 
the schizophrenics out of the hospital and asymptomatic? 

Review of Process 

In appraisal of process one asks if the care rendered meets 
currently accepted standards. In pimple conceptual form this 
represents a feedback loop to insure the program does what it 
claims. Within the broad field of the evaluation of the process, 
it is important to immediately distinguish two relatively different 
methodologies. One is the study of the characteristics of an 
aggregate of a large number of cases to isolate the specific patterns 
of care. For instance one reviews the treatments prescribed for 
various diseases, and diagnostic and laboratory tests needed to 
establish diagnosis and compares these with empirical or norm- 
ative standards to get some appraisal in broad terms of whether 
the care rendered actually meets the standards of quality within 
the field. An alternative technique involves individual case review 
where a committee, usually of physicians, reviews the practice in any 
given case to see whether or not a given patient has received care 
representative of the standards of modern medicine. These 
standards may either be implicit criteria based upon their years of 
experience and general knowledge, or explicit criteria often 
specific for diagnosis. In this latter technique, it is ofton 
extremely valuable to have some method of pre-selecting cases 
which deviate from some norm. How is reviev/ of process 
accomplished? One variable frequently reviewed has been length 
of stay. Multiple regression analysis is a statistical technique 
which surveys cases according to the length of stay for a given 
diagnosis and permits review of those patients whose stay is either 
too long or too short. The implication is not automatically the 
treatment was improper, but rather that there is a higher probability 
of deviant cave being present. 

It is quite clear that these methods of appraisal overlap to a certain 
extent and also exist in a complementary relationship to each other. 
Complete program appraisal is rarely possible simply by end analysis 
but involves relating results to care given. Assessment of process 
may involve assessing progress of a patient to some intermediate end 
point, such as, symptomatic relief, before he leaves the hospital, 

A choice of which of these methods is most applicable depends largely 
upon the state of the art, such as the predictive value of the outcome 
of specific therapies. 
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BASIC UTILIZATION REVIEW PROJECT 



The Basic Utilization Review Program at Yale-New Haven 
Hospital was developed to provide a more sophisticated and 
economical analysis of process. The core of the program is 
a patient classification scheme based on unique combinations of 
diagnostic and demographic characteristics. Empirical norms 
are established for each patient class, and statistically deviant 
cases identified for further review (together with a probability 
sample of non-deviant cases for control purposes). Detailed 
clinical information is abstracted from the patient's chart by 
para-medical personnel and the abstract subjected to evaluation 
according to diagnostic specific pre-established criteria. The 
criteria, constructed by the medical staff of the institution, 
specify the conditions under which the patient should be admitted 
to the hospital, the procedures required by, consistent with, 
and contra-indicated by the diagnosis, and the requirements for 
discharge. The results of the evaluation are then used to 
determine the efficacy of the initial screening and the adequacy 
of the criteria used in the detailed review. This feedback from 
the committee will permit any indicated adjustments to be made 
to the criteria, so that cases which have a high probability of 
needing review will continue to be identified by the screening 
program. Most of the initial screening is now accomplished 
using electronic data processing equipment. Initially developed 
for retrospective review of care for discharged patients, the 
program is presently bein.'j extended to identify cases for con- 
current review. This should permit a more dynamic review 
process, as well as satisfying the various requirements for 
extended duration of stay analysis under medicare and medicaid. 



PROBLEMS IN APPLYING REVIEW IN PSYCHIATRY 

Several problems arise in applying BURP to a psychiatric 
setting, particularly a community mental health center. These 
problems reflect current areas of controversy in the field, such 
as questions about who is sick, what is good care, and what is 
recovery. Lets consider how the project addressed them. There 
is disagreement as to what is a psychiatric illness. The standard 
diagnostic classifications are still basically useful} however, 
problems which were formerly regarded as "social", such as 
prostitution, addiction and illegitimacy are now being called 
"psychiatric". Just what is a psychiatric illness and "who is sick" 
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are lively topics of debate. Furthermore it in difficult to 
develop criteria of quality care solely by diagnosis While 
medical care varies as a function of diagnosis, in psychiatry 
other parameters such as age, social class and the nature of 
the presenting sympton may be crucial. 

The assessment of the functioning of any given individual 
involves the assessment of at least three separate parameters; 
his symptomatology, his social relationships, and some more 
stable abiding characteristics within him that are not time of 
situation dependent which we tend to call character. The same 
outcome may be regarded differently in the eyes of different 
relevant people. The patient, his physician, his family and 
society all maintain different goals. This particular kind of 
problem is not simply the result of a lack of clarity, but is 
the result of the conflicts which lie at the core of psychiatric 
illness itself. As it is difficult to assess a patient at any given 
point within the treatment, if is also difficult to assess his state 
after the treatment, and to answer the question, "what is 
recovery? ", The problems in psychiatry required some 
alterations in approach if we wished to develop a system of 
evaluation which utilized the BURP model but expanded it. 



EXPANDED CONCEPTION OF UTILIZATION REVIEW 



The concept of Utilization Review could be narrowly conceived 
of as merely the study of the allocation of institutional resources 
to ensure their most economic and efficiert use. But we expanded 
the concept to include patient care evaluation. We feel utilization 
review should encompass a review of the following areas: 

1) A study of which patients come to the center and a 
comparison of this population to the total population: served, 

2) A study of what priorities are assigned to various 
patients once in the center. Do certain subgroups, for example, 
on the basis of socio-demographic factors receive differing 
types of treatment? What are the patterns of care? 

3) A study of services actually being provided. Are 
services adequate? What is the quality of care? 

4) A study of outcome of treatment being provided to sec 
whether the services provided are effective. 
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STAGES IN OUR PROGRAM 



Our research program consists of four stages: 

1) Stage one consists of the development of explicit criteria 
for evaluating patient care by peer group panels of mental health 
professionals. These panels are working in different areas: 
schizophrenia— a diagnosis; intake — an institutional process; 
suicide — a symptom; and adolescence — a developmental phase. 
Because of the differing areas being studied, alternate approaches 
to criteria development can be investigated. These arec\s were 
selected in order to assure maximal flexibility in our design. 
Ultimately such panels might also look at some other areas, such 

as field service activities, home visits, community organization, etc. 

One of the heuristic advantages of explicit specification of 
criteria is that it also specifies what information is necessary and 
sufficient to accomplish review. Special attention is required to 
assure that the psychiatric records are sufficiently complete, and 
also retrievable, 

A Data Group within the project has several tasks: 

a) To assist the Mental Health Center in the develop- 
ment of an adequate record system. 

b) To provide information to the clinical panels to 
facilitate the development and validation of criteria 
for case selection and review and to develop tech- 
niques for review of aggregate patterns. This 

has involved coding and analysing the records of 
over 4000 patients in a two year period (1968-1969)* 

c) To assist in the design and evaluation of special 
research studies, 

2) The second stage of the program involves the clinical panels 
testing their criteria for feasibility, reliability and validity by case 
review and selected prospective, follow-up studies, 

3) The third stage involves the implementation of a utilisation 
review mechanism at the CMHC on the basis of integrating the 
various components developed above. This stage would also involve 
an assessment of the actual evaluation process. 
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4) The fourth and final stage of the program involves the 
export of such a system of evaluation and review to other mental 
health centers to evaluate the effectivesnen of the components in 
different settincs and to provide the institutions with the basic 
tools for program evaluation. 



THE "THRU-PUT MODEL" 



The PURE project is funded to accomplish the first two stages 
and some of stage three of the program which we have devised. 
These three main parts to the study in its first year, the clinical 
panels, the data group, the field studies may be visualized as part 
of a "Thru-Put" Model, (See Figure 1) That is, the Mental Health 
Center may be seen as a box which processes input (a portion of 
the population in the community referred as sick) and discharges 
this input in some altered form (increased "health") back into 
the community. The intake panel as one part of its task is 
looking at the selective constraints which are operating so that 
only certain patients come to the center, e,g, such variables as 
distance from the center, community disorganization, types of 
diagnosis, etc. 

This panel is also looking at the adequacy of the intake process, 
from a clinical point of view and developing criteria of what con- 
stitutes an adequate intake. 

The other clinical panels are primarily involved with issues 
involving the scope and quality of care rendered to the patient once 
he is accepted to the Center, These panels are working with the 
data group to describe what patterns of care should Us and what in 
fact they are. They should also be able to delineate what character- 
istics of the patient, such as race or social class, are influencing 
the treatment given. 

The last step in the thru-put conception of the function of a 
mental health center involves return to the community. The two 
field studies, schizophrenia and outpatient field study, address the 
question of the effectiveness of treatment by looking at the patient 
after discharge living in his community. 
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Figure l - "Thru-put" Model 
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GENERA L M >J 1 L OF PROGRAM EVALUATION 

How doet such a research design fit into an overall program 
evaluation. Let ns look at .a general model of program evaluation. 
(See Figure . 1 

Any give p: ogram usually comes into existence with the 
perception of o ne need. Goals are then promulgated by the 
agency which wi hes these goals achieved. Management is hired 
and develops a program. Some group is then hired by the funding 
agency, management* or an outside group hired specifically for 
the task of de rmining whether the goals are being met. This 
reviewing body may be part of the funding agency, management, 
or an outside group hired specifically for the task of evaluation 
and review. 

Ideally the program could be evaluated on the basis of a 
successful ou / 'me, e, g. the prevalence of drug addiction 
decrease in a r ^.lmunity after a drug treatment center is opened. 
Most typically aowever, the full impact of a program is visible 
only after co; ic'crable time has elapsed, and often in areas 
distant from f ho interactional field of the program or direct and 
easy observation, Hence much review focuses on intermediate 
results (i.e. d i 'ixification of an addict) the process itself (i, e, 
psychotherapy lor the addict) and short term outcome (i.e. at 
discharge, adc t is not on drugs). These items are felt to lead 
ultimately to program goals. If there is a strong congruence 
between method of treatment and final outcome, this is sufficient. 
Where the congruence is limited, it is necessary to do special 
studies to look at whether the program is meeting its stated goals. 

The reviewing group has to develop mechanisms for selecting 
from the abundance of data on goals, the program, and the outcome 
of the program those aspects which it feels will yield information 
useful for review and for feedback into the system. This decision 
i6 heavily dependent on the type and character of data which can 
be obtained in analyses. 

In the final step of our model the reviewing process yields 
information which can be fed back to any of the previous stages to 
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Figure Z 



GENERAL MODEL OF PROGRAM EVALUATION 
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modify the system. These program goals, the program itself, 
the selection process, or the review process itself might be 
modified. It is clear that to change any of these areas will have 
wide ramification in changing all areas. 

As an example pertinent to community mental health, let's 
look at the problem of out-patient services. Suppose the reviewing 
group find few patients being treated in an outpatient department 
and long waiting lists. Farther they find patients returning with 
greater frequency than they do in comparable institutions. This 
information would have implications for organizational structure, 
allocation of resources, examination of methods of treatment, 
and changes in the data system to provide better records for 
examination. Also outcome studies would be devised to look for 
change. All of these changes would make the group again look 
at its methods for selection for review. This would help decide 
whether information is most usefully obtained by examining patterns 
of care of aggregates of patients or by single case review of those 
patients whose individual care seems to deviate from the norm. 

Having now described our overall project (panels, data group, 
field studies), how does this structure fit into our conceptual model 
of evaluation? By and large the panels are covering the area of 
examining those aspects of the program which relate to patient care. 
They are developing criteria of good care, selection mechanisms, 
and are testing the utility, validity and reliability by a series of 
research, projects. The field studies are looking at program 
outcome. The intake panel is examining the nature of program 
input. 

The Psychiatric Utilization and Review Project has been 
involved with the top portion of our model. Within the next few 
months, in conjunction with the administrative staff of the 
Connecticut Mental Health Center, a reviewing process will be 
started which will provide the last step in our model - a step 
necessary for a system of feedbacks to all levels, both in the 
project and in the Center. We will examine the impact of this on 
the Center, 

It is important to point out that this entire system focuses on 
those program goals related to patient care. It does not attempt 
to do evaluation of those aspects of the program such as the 
success of community participation on making policy. 
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Patient care evaluation encompasses utilization review and 
patient care evaluation is a basic step in program evaluation. 

The focus on patient care evaluation is especially pertinent since 
a fundamental mandate of the community mental health center has 
been the provision of services, especially for those who previously 
received no care or inadequate care. 

We realize the review process cannot by fiat impose a set of 
standards on an institution where practices for ideologic or exper- 
imental reasons may differ from norms. Rather it is the function 
of review in such instances to ascertain whether the care given 
corresponds to the manifest goals of a given service and to delineate 
more clearly areas where differing practices make comparisons 
fruitful. As the reviewing mechanism becomes sufficiently sensitive 
to detect not only cases which deviate from desired norms, but also 
cases where desired treatment norms have been met but whose 
outcome is unsatisfactory, areas for further investigation will be 
brought into view. It becomes clear that in psychiatry, review and 
research are potentially synonomous in certain areas. In a field 
which is known for its dearth of effective outcome studies, and 
which has difficult methodology associated with measures of outcome^ 
it is not surprising or unexpected that the initial stages of a review 
project should be found confronting the issues which have been 
plaguing other kinds of research in the fields for many decades. 

With proper attention to the breadth of material to review, and to 
the many levels of feedback necessary, a review program will 
result both in increased quality of care and clarification of issues 
now conceptually unclear. 





